QVAC-27164 feat[api|bc]: support EmbeddingGemma 2 multimodal embeddings by maxim-smotrov · Pull Request #5464 · tetherto/qvac

maxim-smotrov · 2026-10-30T02:00:12Z

🎯 What problem does this PR solve?

SDK doesn't provide support for EmbeddingGemma 2 for multi-modal embeddings. The addon already added support in #5237.

📝 How does it solve it?

Add ordered inputs batches with text, file-path media, and base64 media while preserving the existing text calls.
Resolve the projector through the model lifecycle and expose context, media, and projector settings.
Update the wire contract, Python and Kotlin bindings, examples, documentation, and e2e catalog.
Keep this capability library-only. CLI OpenAI-compatible embeddings remain text-only.
Registry entries are tracked separately in QVAC-27164 feat[mod]: add EmbeddingGemma 2 to the registry #5463.

🧪 How was it tested?

Full local inference, SDK, and e2e build passed.
Inference lint/type checks and focused Bare tests passed, 13 tests and 67 assertions.
SDK lint/type checks, unit tests, and contract freshness checks passed.
Python generation check and Kotlin contract/JVM tests passed.
E2e formatting, type checks, and catalog checks passed.
Real-model e2e run on macOS.

💥 Breaking Changes

The generated Kotlin and Python EmbedRequest types now represent two alternatives. EmbedTextRequest requires text and forbids inputs. EmbedInputsRequest requires inputs and forbids text.

A single object with optional text and inputs would keep the old constructors, but its Zod refinement requiring exactly one field would not appear in the exported JSON Schema. Generated clients could then accept both fields or neither. With the union, Kotlin callers must select a variant with a required input field, and Python rejects both or neither during model validation, before RPC. The cost is the generated-client migration below. Existing QvacEmbeddings.embed convenience methods remain unchanged.

The TypeScript embed() overloads preserve precise result types: text: string returns one vector, while text: string[] and inputs return a batch of vectors. These overloads do not cause the generated-client migration; the request schema union does. Existing TypeScript text calls remain supported.

Kotlin

Direct construction of EmbedRequest now requires a request variant wrapper. The text value type is renamed from EmbedRequestText to EmbedTextRequestText.

BEFORE:

import io.tether.qvac.sdk.generated.schema.EmbedRequest
import io.tether.qvac.sdk.generated.schema.EmbedRequestText

val request = EmbedRequest(
    modelId = "m",
    text = EmbedRequestText.Variant1("hi"),
    requestId = "embed-1",
)

AFTER:

import io.tether.qvac.sdk.generated.schema.EmbedRequest
import io.tether.qvac.sdk.generated.schema.EmbedTextRequest
import io.tether.qvac.sdk.generated.schema.EmbedTextRequestText

val request = EmbedRequest.EmbedTextRequest(
    EmbedTextRequest(
        modelId = "m",
        text = EmbedTextRequestText.Variant1("hi"),
        requestId = "embed-1",
    ),
)

For text batches, use EmbedTextRequestText.Variant2(texts) inside the same wrapper. For multimodal batches, use EmbedRequest.EmbedInputsRequest wrapping an EmbedInputsRequest. When inspecting a request typed as EmbedRequest, select the variant and read its fields through .value.

The existing QvacEmbeddings.embed(modelId, text) and embed(modelId, texts) convenience methods remain supported and need no caller migration.

Python

The existing keyword constructor still works. EmbedRequest is now a Pydantic RootModel containing the selected request variant, so direct .model_id, .text, and .request_id access must move under .root.

BEFORE:

from tetherto.qvac_sdk import EmbedRequest

request = EmbedRequest(model_id="m", text="hi", request_id="embed-1")
model_id = request.model_id
text = request.text
request_id = request.request_id

AFTER:

from tetherto.qvac_sdk import EmbedRequest

request = EmbedRequest(model_id="m", text="hi", request_id="embed-1")
model_id = request.root.model_id
text = request.root.text
request_id = request.root.request_id

For the inputs variant, read request.root.inputs after selecting that variant. Continue passing the outer request to embed(transport, request); do not replace it with request.root. Wire serialization remains flat, with no root or value field added to the RPC payload.

🔌 API Changes

Load the model with modelConfig.projectionModelSrc, then pass text and media inputs:

const { embedding } = await embed({
  modelId,
  inputs: [
    'task: search result | query: an elephant',
    [{ media: { type: 'filePath', value: '/path/to/elephant.jpg' } }]
  ]
})

The result contains one vector per input.

github-actions · 2026-10-30T17:07:28Z

License compliance — clean

No new dependency license findings in this PR.

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

./docs/website
./packages/asr-ggml/benchmarks/server
./packages/fabric/test/integration
./packages/llm-llamacpp/benchmarks/performance
./packages/llm-llamacpp/benchmarks/server
./packages/inference-addon-cpp/mobile
./packages/sdk-e2e
./packages/embed-llamacpp/benchmarks/performance
./packages/embed-llamacpp/benchmarks/server
./packages/vla-ggml/sim/server
./.github/actions/release-merge-guard
./.github/bare-toolchain

github-actions · 2026-10-30T18:11:53Z

Review Status

Current Status: ❌ PENDING
Approvals so far: Member: 2

Pending reviews: Needs 1 Management or Team Lead.

greptile-apps · 2026-11-02T11:49:56Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issue was found.

Summary

Adds library-only text, image, and audio embeddings with EmbeddingGemma 2 while preserving existing text calls.

Embedding calls can now turn ordered text and media into vectors.
SDK clients can choose text or media batches with typed requests.
EmbeddingGemma 2 now has cross-platform checks for media vectors.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Load embedding model] --> B[Resolve model and optional projector]
  B --> C[Create and load addon]
  D[Embed text or ordered inputs] --> E[Admit one request per model]
  E --> F[Check projector before reading media]
  F --> G[Prepare text and media bytes]
  C --> H[Run addon]
  G --> H
  H --> I[Normalize vectors and return results]
  I --> J[Finish cancellation cleanup]
  J --> K[Release request slot]

_{Reviews (21) · Last reviewed commit: "Merge branch 'main' into feat/sdk-embedd..." · Reviewed by Greptile}

github-actions · 2026-11-03T15:10:40Z

QVAC E2E — base run recorded

Base: run 41661064462 · suite smoke · sha c367ca9

Platform	Passed	Total
`android/android`	187	213
`desktop/linux`	213	213
`desktop/macos`	210	213
`desktop/windows`	213	213
`ios/ios`	183	213

All recorded platforms are green — nothing for test-e2e-rerun-failed to re-run.

github-actions · 2026-11-03T15:10:55Z

QVAC E2E — tests changed by this PR

This smoke run additionally covers 89 test(s) that are not tagged smoke, adding roughly 59.1 min.

Catalog: 624 tests · smoke: 124 · touched by this PR: 129 (40 already in smoke)

Changed file	Matched via	Tests
`packages/inference/src/handlers/assess-model-fit`	sdk handler	26
`packages/sdk/src/client/api/embed.ts`	sdk api	85
`catalog/embeddinggemma2-tests.ts`	definitions (whole file)	4
`catalog/platform-skips.ts`	import graph	18
`catalog/test-definitions.ts`	definitions (whole file)	18
`runners/js/shared/executors/embeddinggemma2-executor.ts`	executor pattern	4

89 added test id(s)

model-load-llm-load-mode-none
model-load-llm-legacy-no-mmap-rejected
model-load-llm-prompt-cache-keys
model-load-llm-cache-checkpoints-out-of-range-rejected
model-load-embedding-rank-pooling-rejected
model-reload-llm
embed-long-text
embed-empty-text
embed-similarity
embed-unicode
embed-very-short
embed-code-snippet
embed-multilingual
embed-special-chars
embed-numbers-only
embed-semantic-similarity
embeddinggemma2-image-retrieval
embeddinggemma2-audio-deterministic
embeddinggemma2-text-parity
embeddinggemma2-reject-corrupt-media
vector-index-remove-contains
vector-index-write-load
sharded-model-hash-validation
sharded-model-backward-compatibility
sharded-model-progress
sharded-model-resume
sharded-model-cancellation
sharded-model-batch-inference
sharded-model-long-text-inference
sharded-model-llm-completion
sharded-model-llm-reload
sharded-model-llm-missing-shards
http-sharded-embed-progress
http-archive-embed-load
http-archive-embed-progress
http-archive-embed-inference
model-fit-probe-llm
model-fit-probe-embeddings
model-fit-probe-parakeet
model-fit-probe-whisper
model-fit-probe-bci
model-fit-probe-tts-chatterbox
model-fit-probe-audiogen
model-fit-probe-diffusion
model-fit-probe-diffusion-esrgan
model-fit-probe-upscaler
model-fit-probe-ocr
model-fit-probe-translation
error-invalid-response-type
error-model-load-failed
… and 39 more

These changed files map to no test, so nothing was added for them — review by hand if they affect behaviour:

packages/inference/src/plugins/builtin/llamacpp-embedding
catalog/resource-table.ts
catalog/resources/resource-table.json
catalog/resources/skip-matrix.json
runners/js/desktop/consumer.ts
runners/js/electron/consumer.ts
runners/js/mobile/consumer.ts

github-actions · 2026-11-03T15:12:06Z

QVAC E2E — `windows` — ✅ all tests passed (213/213, 427s)

Config: suite=smoke · filter=(none) · exclude=(none)
Inference: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/inference@0.23.0 · Test-suite: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/test-suite@0.13.0
View run · Artifacts: reports

github-actions · 2026-11-03T15:12:07Z

QVAC E2E — `linux` — ✅ all tests passed (213/213, 399s)

Config: suite=smoke · filter=(none) · exclude=(none)
Inference: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/inference@0.23.0 · Test-suite: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/test-suite@0.13.0
View run · Artifacts: reports

github-actions · 2026-11-03T15:12:08Z

QVAC E2E — `macos` — ✅ all tests passed (210/213, 437s)

Config: suite=smoke · filter=(none) · exclude=(none)
Inference: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/inference@0.23.0 · Test-suite: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/test-suite@0.13.0
View run · Artifacts: reports

github-actions · 2026-11-03T15:12:28Z

QVAC E2E — `ios` — ✅ all tests passed (183/213, 1192s)

Config: suite=smoke · filter=(none) · exclude=(none)
Inference: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/inference@0.23.0 · Test-suite: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/test-suite@0.13.0
Device pool: iPhone 17 Pro - iOS 26 — ☀️ highly available · Billing: UNMETERED
View run · Artifacts: reports · Device Farm logs

github-actions · 2026-11-03T15:12:29Z

QVAC E2E — `android` — ✅ all tests passed (187/213, 3942s)

Config: suite=smoke · filter=(none) · exclude=(none)
Inference: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/inference@0.23.0 · Test-suite: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/test-suite@0.13.0
Device pool: Google Pixel 9 Pro - Android 15 — ☀️ highly available · Billing: UNMETERED
View run · Artifacts: reports · Device Farm logs

Signed-off-by: Maksim Smatrou <maxim.smotrov@itrexgroup.com>

…ction

lauripiisang

The way we do e2e is a bit different now so the implementation should be changed to match there, plus the schemas could use titles so they get better naming in generated code.

Signed-off-by: Maksim Smatrou <maxim.smotrov@itrexgroup.com>

maxim-smotrov marked this pull request as ready for review October 30, 2026 18:11

maxim-smotrov requested review from a team as code owners October 30, 2026 18:11

maxim-smotrov force-pushed the feat/sdk-embeddinggemma2 branch from 539e642 to 9c4b45b Compare October 30, 2026 18:27

greptile-apps Bot reviewed Nov 2, 2026

View reviewed changes

Comment thread packages/inference/src/plugins/ops/embed.ts Resolved

Comment thread packages/sdk-e2e/runners/js/shared/executors/embeddinggemma2-executor.ts Resolved

github-code-quality Bot found potential problems Nov 2, 2026

View reviewed changes

Comment thread packages/inference/test/embed-plugin.test.ts Fixed

maxim-smotrov changed the title ~~QVAC-27164 feat[api]: support EmbeddingGemma 2 multimodal embeddings~~ QVAC-27164 feat[api|bc]: support EmbeddingGemma 2 multimodal embeddings Nov 2, 2026

maxim-smotrov force-pushed the feat/sdk-embeddinggemma2 branch 2 times, most recently from cb9fb86 to c367ca9 Compare November 3, 2026 14:19

maxim-smotrov added the test-e2e-smoke Triggers smoke e2e test suite [Currently SDK-only] label Nov 3, 2026

maxim-smotrov had a problem deploying to release November 3, 2026 15:12 — with GitHub Actions Failure

maxim-smotrov temporarily deployed to release November 3, 2026 15:12 — with GitHub Actions Inactive

maxim-smotrov temporarily deployed to release November 3, 2026 15:22 — with GitHub Actions Inactive

maxim-smotrov temporarily deployed to release November 3, 2026 15:24 — with GitHub Actions Inactive

maxim-smotrov temporarily deployed to release November 3, 2026 15:25 — with GitHub Actions Inactive

maxim-smotrov temporarily deployed to release November 3, 2026 15:26 — with GitHub Actions Inactive

maxim-smotrov added 8 commits November 4, 2026 16:17

chore: use EmbeddingGemma 2 model constants

7f31419

Signed-off-by: Maksim Smatrou <maxim.smotrov@itrexgroup.com>

fix(docs): move multimodal embeddings to development line

e375b95

fix: honor embedding context size in model fit estimates

efb1236

fix: validate embedding load config before resolving models

da248da

test: remove redundant embedding lifecycle coverage

82ef777

fix(inference): isolate embedding cancellation and preserve busy reje…

1ddaf2e

…ction

fix(inference): reject embedding media without a projector

706857d

fix(inference): reject empty embedding text parts

e9a8bc4

maxim-smotrov force-pushed the feat/sdk-embeddinggemma2 branch from 2c66681 to e9a8bc4 Compare November 4, 2026 15:17

dev-nid previously approved these changes Nov 5, 2026

View reviewed changes

lauripiisang reviewed Nov 5, 2026

View reviewed changes

Comment thread packages/sdk/test/embed-client.test.ts Resolved

lauripiisang requested changes Nov 5, 2026

View reviewed changes

fix(sdk-python): enforce forbidden request fields

5ff6c83

maxim-smotrov dismissed dev-nid’s stale review via 5ff6c83 November 5, 2026 13:16

maxim-smotrov added 8 commits November 5, 2026 17:17

test: make EmbeddingGemma 2 e2e cases declarative

179b1fc

Signed-off-by: Maksim Smatrou <maxim.smotrov@itrexgroup.com>

fix: preserve nested request union errors

56e970d

Signed-off-by: Maksim Smatrou <maxim.smotrov@itrexgroup.com>

fix: name generated embedding input types

586fc09

Signed-off-by: Maksim Smatrou <maxim.smotrov@itrexgroup.com>

test: isolate embed client mock behavior

38d1121

Signed-off-by: Maksim Smatrou <maxim.smotrov@itrexgroup.com>

test(sdk-e2e): add Kotlin embedding vector assertions

fb06a66

Signed-off-by: Maksim Smatrou <maxim.smotrov@itrexgroup.com>

fix(sdk-python): expose EmbedRequest to type checkers

3d2d143

Signed-off-by: Maksim Smatrou <maxim.smotrov@itrexgroup.com>

fix(sdk-python): forward schema exports to type checkers

443e7c0

Signed-off-by: Maksim Smatrou <maxim.smotrov@itrexgroup.com>

test(sdk-python): expect embed type in serialized requests

1107abf

greptile-apps Bot reviewed Nov 5, 2026

View reviewed changes

Comment thread packages/sdk-python/tests/test_embed.py Resolved

fix: default multimodal example to sample media

81430ca

Signed-off-by: Maksim Smatrou <maxim.smotrov@itrexgroup.com>

lauripiisang approved these changes Nov 6, 2026

View reviewed changes

dev-nid approved these changes Nov 6, 2026

View reviewed changes

Merge branch 'main' into feat/sdk-embeddinggemma2

3fdf3f2

gianni-cor approved these changes Nov 6, 2026

View reviewed changes

gianni-cor merged commit 54d496a into main Nov 6, 2026
51 checks passed

gianni-cor deleted the feat/sdk-embeddinggemma2 branch November 6, 2026 09:20

Conversation

maxim-smotrov commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎯 What problem does this PR solve?

📝 How does it solve it?

🧪 How was it tested?

💥 Breaking Changes

Kotlin

Python

🔌 API Changes

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

License compliance — clean

NOTICE presence (advisory)

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review Status

Uh oh!

greptile-apps Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QVAC E2E — base run recorded

Uh oh!

github-actions Bot commented Nov 3, 2026

QVAC E2E — tests changed by this PR

Uh oh!

github-actions Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QVAC E2E — windows — ✅ all tests passed (213/213, 427s)

Uh oh!

github-actions Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QVAC E2E — linux — ✅ all tests passed (213/213, 399s)

Uh oh!

github-actions Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QVAC E2E — macos — ✅ all tests passed (210/213, 437s)

Uh oh!

github-actions Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QVAC E2E — ios — ✅ all tests passed (183/213, 1192s)

Uh oh!

github-actions Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QVAC E2E — android — ✅ all tests passed (187/213, 3942s)

Uh oh!

Uh oh!

lauripiisang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

maxim-smotrov commented Oct 30, 2026 •

edited

Loading

github-actions Bot commented Oct 30, 2026 •

edited

Loading

github-actions Bot commented Oct 30, 2026 •

edited

Loading

greptile-apps Bot commented Nov 2, 2026 •

edited

Loading

github-actions Bot commented Nov 3, 2026 •

edited

Loading

github-actions Bot commented Nov 3, 2026 •

edited

Loading

QVAC E2E — `windows` — ✅ all tests passed (213/213, 427s)

github-actions Bot commented Nov 3, 2026 •

edited

Loading

QVAC E2E — `linux` — ✅ all tests passed (213/213, 399s)

github-actions Bot commented Nov 3, 2026 •

edited

Loading

QVAC E2E — `macos` — ✅ all tests passed (210/213, 437s)

github-actions Bot commented Nov 3, 2026 •

edited

Loading

QVAC E2E — `ios` — ✅ all tests passed (183/213, 1192s)

github-actions Bot commented Nov 3, 2026 •

edited

Loading

QVAC E2E — `android` — ✅ all tests passed (187/213, 3942s)